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Abstract 

Despite the increasing importance of language testing, technology design matters related to testing have 
hardly been addressed in academic research, yet not enough has been written on test-taking anxiety 
related to computerized speaking tests. Although students nowadays are digital natives, in our experience 
they seem to experience a significant level of anxiety when facing Internet-based oral. This research 
addresses an observational study of 67 students at the University of Alcala who took a computer-delivered 
exam. The study concludes that despite students growing up with computers in their lives, we are far from 
invisible interfaces and independence when it comes to the way an exam is delivered. The results of this 
research study can shed light on what needs to be revised in computer or internet speaking tasks in high- 
stakes language tests 


Resumen 

A pesar de la creciente importancia de los examenes de idiomas, los asuntos relacionados con el diseno 
tecnolégico aplicados a la realizacién de estas pruebas cuando se realizan por ordenador apenas se han 
abordado en la investigaci6n académica, y no se han publicado suficientes estudios sobre la ansiedad del 
estudiante a la hora de realizar pruebas orales de idiomas en frente de un ordenador. Aunque podemos 
considerar que los estudiantes de hoy en dia son nativos digitales, la experiencia nos muestra que estos 
manifiestan un importante nivel de ansiedad a la hora de realizar pruebas orales de idiomas a través de 
Internet. En este articulo se analizan los resultados de un estudio observacional realizado con una 
muestra de 67 estudiantes de la Universidad de Alcala que hicieron una prueba de idioma por ordenador. 
El estudio concluye que, a pesar de que los estudiantes crecen con los ordenadores y estan 
familiarizados con los mismos, aun estamos lejos de conseguir interfaces invisibles y de la independencia 
en cuanto a la forma en que se entrega un examen. Los resultados de este estudio de investigacién 
pueden arrojar luz sobre lo que necesita revisarse a la hora de disenar examenes de idiomas que se 
realizan a través del ordenador 
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1. Introduction 


Computer-based languages tests are common in education today (Chapelle and Voss, 2016; 
Yu & Zhang, 2017). Recent work in this area has looked at the use of devices such as mobile 
phones (Garcia Laborda, Magal Royo, Litzler and Giménez Lopez, 2015) or tablet PCs (Garcia 
Laborda, Magal Royo and Bakieva, 2016). Excellent reviews have analysed some of the 
reasons for their overwhelming use. Some of the advantages related to it are their reduced cost, 
rapidness to mark tests and the inclusion of contextual videos to help students understand what 
they need to do (Garcia Laborda, 2007). Today, it is hard to understand the significant 
advances in high-stakes language tests such as tests for citizenship, placement of international 
students, university entrance examinations and many more without the influence of the facility of 
delivery of computer or web tests. Although computer language tests differ from web-based 
tests in their design and retrieval, for practical purposes both types have been considered in this 
paper to be similar enough not to make a distinction. 


Despite a number of technological implementations (Labi, 2006), Garcia Laborda, Magal Royo 
& Carrasco (2010) considered a few years ago that computer-based testing would completely 
replace pen and paper in just a few years (Dooey, 2008; Tahmasebi & Rahimi, 2013). However, 
reality has proved an unexpected co-existence of both ways of delivery. Thus, the question is 
why these pen-and-paper tests are still among us. Some factors that may affect this presence 
are related to the high age of test takers, the incapability of places in the third world to have 
good Internet access, obsolete hardware in some schools, wi-fi defective or weak connections. 
However, these may not actually explain the case for traditional exams in developed countries 
(at least, not always). Thus, it is necessary to get to know what the constraints are in foreign 
language testing. This paper addresses the observations done in the OPENPAU research 
project on computer-based University Entrance Examination between 2011 and 2016. 


2. Literature review 


Language tests have been traditionally contrasted against pen-and-paper foreign language 
tests. Khoshsima, Hosseini & Toroujeni (2017) consider that in most cases the interest has 
been placed on converting traditional paper tests into computer delivered ones. However, they 
believe that computer-based language tests may not measure “Foreign Language only’ but 
other aspects as well. Obviously, if this is true the validity of the exam would be jeopardized. 
According to Schneberger, Amoroso & Durfee (2007) the drawbacks introduced by the use of 
computers can be overcome by training and an adequate knowledge of the test. Garcia 
Laborda, Litzler, Amengual Pizarro, Garcia Esteban & Otero de Juan (2017) observed in a 
study on the APTIS test that when students know well what they are expected to do in a test 
and are very familiar with the candidate’s instructions book, their scores improve. Mei, Brown & 
Teo (2018) suggest that there is an increasing acceptance of the use of technology in the 
English as a Foreign Language classroom and that leads to better training in assessment 
through computers. Thus, as it can be seen, knowing the skills that can be more accessible is a 
key issue. Coniam (2006; 2009) describes how to write listening items for language tests. He 
also implies that listening, especially with multiple choice questions, is more accessible than 
other methods. The use of multiple-choice items facilitates delivery especially in adaptive testing 
(He & Min, 2017) but may well put the validity of a four-skill language test at stake since multiple 
choice language knowledge in real life does not exist. Besides, according to Odo (2012) 
familiarity with computers facilitates test performance. This is especially important because the 
increasing use of adaptive language tests is due to the fact that they make it easier to create 
new items and this can be done at a lower cost. 


Garcia Laborda & Magal Royo (2009) researched the facility that senior teachers had to adapt 
to computer-based tests. They found that even older teachers would be accepting to orientate 
their teaching towards computer-based exams if they are trained properly. In a three-month 
training course, they observed that teachers are willing to respond positively if they are shown 
the functional and beneficial aspects of using computers that way. 
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Fox & Cheng (2015) studied some of the issues that unsuccessful IB TOEFL test takers 
referenced as part of their personal experience with the test. Among the most significant 
aspects, they mention speediness, test anxiety, and test preparation. Naturally, these aspects 
have been found in current literature on the topic and Fox & Cheng include them in their paper. 
However, they fail to explain the issues that have to be specifically considered. Like other 
papers, this one also ignores the important issue of class observation done by researchers. 
Another important study was done by Zhan & Wan (2016) who report on students’ experience 
when preparing for a listening and speaking test (CELST). Thus, it was also important to get to 
know how teachers prepare their students for computer tests. Garcia Laborda, Magal Royo & 
Enriquez Carrasco (2010) as well as Liu & Kleinsasser (2015) suggest that what seems to be a 
crucial aspect is teachers becoming aware of technology as well as getting the training to 
specifically address the necessary computer skills to their students through classroom activities. 
Gebril & Eid (2017) suggested a number of activities to prepare for online high-stakes tests. 
Teachers also mentioned that high-stakes tests may lead to negative washback (Messick, 1996; 
Popham, 2001) and put the emphasis on training techniques and test skills rather than focusing 
on learning the language. 


As seen in this review, it would be desirable to inform teachers about the most significant 
aspects to consider when training students for language tests based on the difficulties they have 
when taking them and how to solve some of the problems that derive from it. This paper intends 
to respond to these two questions. 


3. Method 
3.1. Participants 


The sample consisted of 67 students. Observations were done in three high-schools in Madrid 
and two in Guadalajara (Spain). Data collection was done on a research diary and, additionally, 
some of the parts of the research were videorecorded. The schools were categorized as 
average in the compulsory University Entrance Examination English scores. 

This observational study intends to address two main questions from the researcher's 
perspective: 


1) What are the difficulties that test takers have when taking an online test? 
2) How can some of these problems improve? 


This research involves the presence of the researcher and his observation may bias the 
research. A special effort has thus been made to verify the observations against the recorded 
videos of the observed experiences. Moreover, despite this potential bias, it has been especially 
important to understand and record the context and the interaction with the test and the 
computer. This observational experience will open new questions and research for future 
experiences that have not been implemented yet. The methodological approach hereby 
followed corresponds to Bryman (1988). The observations correspond to the OPENPAU project 
(2011-2016). 


3.2. Procedure 


As already said, the researcher used observational methods to collect ground notes on 
students’ use and interaction. Each exam session lasted two hours, usually in two shifts due to 
the number of computers or the wifi capacity when using tablet PCs. During each visit, students 
were identified by a number code in order to track them, although this information may not be 
necessary for this paper. Before the visit, the research team was unaware of the situation in 
each high-school. Only the technicians of the research team were able to see the hardware and 
connections in advance. Although other publications have informed about the results of the 
OPENPAU, this is actually the first time that constraints are specifically addressed. 
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The researcher operationally annotated different modes of interaction despite the number of 
occurrences in the research. Annotations were classified according to the different skills. 
However, the most significant observations refer to the speaking section. 


4. Results and discussion 


Testing requires a state of slight positive anxiety. In the observations, two types of attitudes 
were observed. While most students showed relative anxiety, a few actually evidenced high 
anxiety because of certain discomfort while taking the test. This discomfort was due to the 
following factors: 


e Time pressure. This feeling was emphasized by the presence of a clock in the 
interface (like in all computer-based tests). This is obviously very different from pen- 
and-paper tests where the clock may be in front of test takers and increase their 
anxiety (Roever, 2006; Liu et al., 2010; Namdar & Bagheri, 2012; David, 2016). A 
second matter in this sense is that in a traditional exam students generally do not 
need to “pack’ their oral responses in a given time. We observed, for instance, that 
the guide to a number of the Cambridge Suite exams recommend not only what and 
how to respond but to adjust the response to the given time. Thus, students have the 
feeling that they have to keep speaking until the time is reached. 


e Lack of interaction. This has been mentioned in a number of papers such as the one 
published by Qian (2009) but we found that most students do not hesitate to interact 
with semi-communicative videos especially when addressing descriptions. In another 
research study done with French students in the Pennsylvania State Examinations at 
State College High School in 2013, the researcher observed that when the input of the 
test task is not visually supported, results tend to be more discrete. 


e Problems with body language. We even observed students using their body language 
(Chen & Wang, 2008). Therefore, hesitation or “emptiness” may well be associated to 
personality styles. In a different research study with the APTIS test, it was considered 
that the least and the most extroverted students were more fluent and used more 
body language when doing descriptions in the speaking section. 


e Feeling unheard. Additionally, it has been observed that in paired interviews there is 
considerable support and interaction not only verbally but also through body language 
which is missing in online testing (Ockey, 2009). A number of students mentioned that 
they lack the motivation to answer to questions prompted by a video with little 
expectations to communicate meaningfully. 


e “Dehumanization” (nobody could care about what they said). As in the previous point, 
the lack of communication is connected to dehumanization where there can be 
repulsion towards what they are doing. In extreme situations research suggests that 
this fact could lead to lowering their scores. 


e Prompts cannot replace human presence. Although a number of studies have 
intended to provide evidence that videos in computer-based testing can help to 
contextualize the prompts, this fact depends on the testee’s personality. 


After the observations, the researcher, however, observed some positive aspects which could 
actually be related to positive reactions to computer language testing. Again, it was perceived 
that the main trait that could actually make a distinction between the previous and the following 
reactions would be the test taker’s personality. Among the interesting reactions it was observed: 


e Body language in one-way interaction. Some students, who were mostly considered 
as introverted by their teachers, evidenced a great capacity to interact with the 
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computer. It was even observed that around 15% did some kind of body language 
when addressing the speaking questions. 


e Relation with competence (lower competence demands more interaction). This body 
language was also observed in students with lower competence who may need to rely 
on body language. 


e High adaptability to computers. It has been observed in literature that some people 
are more adaptable to interact with computers than others. Some of the test takers 
proved to be more capable to adapt to this kind of tests than others. However, 
although this could be a natural thing, it is important to find the features that make 
some students more prone to interact with computers in language tests and those that 
actually do the opposite. 


e Despite the “empty responses” students are able to synthesize and be able to 
articulate good monologues. The researcher observed that, all in all, most students 
with at least a B1 competence are able to articulate the adequate responses and even 
monologues. 


Although so far we have addressed significant problems that may threaten students’ final 
scores, we observed quite a positive situation among high schoolers. That may well indicate 
that we are not far from that transparent and normalized interface which may have little or no 
effect on how students perform. This may be due to the fact that students are becoming more 
familiar with online language tests and also because technology involves many daily 
interactions. Additionally, technology in language recognition may make this easier even in the 
future (Chiu, Liou & Yeh, 2007). Therefore, students are getting used to interacting with avatars, 
videos and digital prompts as well as using online applications to trigger their speaking skills 
(Yen, Hou & Chang, 2015). Test designers should thus develop prompts similar to the most 
common ones found in digital communication (Yang, Miller & Bai, 2011; Yang, Gamble & Tang, 
2012; Andujar-Vaca & Cruz-Martinez, 2017). 


It was also observed that some semi-communication can be established with our current ways 
to assess through computer delivered language tests. This is currently presented in many 
language lab activities. It was also considered that there is still some scarcity of online 
published materials that have been specifically designed for test training. 


5. Conclusions 


The results of this observational study have shown that even though students nowadays are 
more familiar with new technologies and these are indeed part of their everyday lives, when 
taking a test anxiety still seems to be an issue if this is done in front of a computer. As stated 
above, the advantages of online testing are obvious, such as the possibility to spread them all 
over the world and obtain results more quickly. However, if we want our students to improve 
their performance when using computers to take tests, we need to make sure that they are 
completely comfortable with it after having practiced for a long time with the format of the test. 
They need to be sure that they will not waste any time learning how to use the tool or reading 
instructions and that they can start working on the contents of the test as soon as they have it in 
front of them. This is why teachers need to be prepared to be able to train their students 
properly on the use of these tests. The demands and speed of technology of the world of 
education nowadays demand a critical teacher that is able to combine the teaching of language 
content with the digital competencies that students need to develop to successfully meet the 
requirements of online testing. Therefore, the crucial aspect of teachers’ preparation is not just 
teaching students the language contents they need for the test but making them aware of the 
skills they need to develop to specifically address the challenges of online testing. This will 
result in fewer problems related to time pressure and lack of interaction and will help students 
realize that not having the presence of a human when they speak does not mean that what they 
are saying is meaningless or empty. 
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This research has addressed an observational study of 67 students at the University of Alcala 
who took a computer-delivered exam. The observations were part of the OPENPAU research 
project on computer-based University Entrance Examination between 2011 and 2016. The 
questions that the research study intended to give an answer to were related to the difficulties 
that test takers had when taking an online test and how some of the problems that derive from it 
could be solved. Results show that the anxiety experienced by students when taking a 
computer-based test was linked to factors such as time pressure, lack of interaction, problems 
with body language, feeling unheard and lack of human presence to interact with. However, 
some positive aspects were also observed since many students were able to adapt to the 
computer-based test and successfully interact with it. The variation in reactions seems to 
depend on the test taker’s personality. This may well be a sign of progress in web-based testing 
since students are becoming more used to interacting with technology. Moreover, results help 
teachers understand what factors they need to work on in order to facilitate students’ 
experience when taking an online test and more online materials need to be created in order to 
support training in this regard. 


However, we are aware that this research study is clearly limited by the sample and the 
circumstances in which the observations were done. Thus, further research is necessary. One 
of the issues that needs to be developed as contrasted to real life is how to achieve a “flow’ 
state of mind in an online test (in fact, either traditional or online tests) (In’nami, 2006; Aydin, 
2013; Bayat, Jamshidipour & Hashemi, 2017; Valencia Robles, 2017) despite the time and 
other issues constraints (Behnam, Jenani & Ahangari, 2014). The reduction of high negative 
anxiety could facilitate student’s performance in most cases. This reduction of negative anxiety 
should be a goal in any exam to achieve the student’s full potential and thus currently needs 
specific training (Wang & Chang, 2011). Another significant potential study would be that of 
personality in online language testing Lai, 2010) which could also relate to age and socio- 
economic status, although the latter is gradually disappearing as a differentiating factor. All in 
all, this paper has addressed the researcher’s observations. However, larger studies are still 
necessary since new issues will appear as technology evolves. In this sense, this paper may 
serve to trigger future studies and serve as food for thought for teachers, researchers and 
administrators. 
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